fix: add file-size guard to prevent memory spikes from oversized files (APP-4519)#11410
Closed
warp-dev-github-integration[bot] wants to merge 1 commit into
Closed
Conversation
APP-4519) Add a 50 MiB file-size limit to the editor buffer loading pipeline to prevent multi-gigabyte memory spikes caused by loading very large files into SumTree buffers and tree-sitter parsers. Root cause: GlobalBufferModel::populate_buffer_with_read_content loaded arbitrarily large file content into Buffer::replace_all, which triggers SumTree<BufferText>::append_str allocations and SyntaxTreeState::parse_text. The pprof heap profile showed 8.07 GB through Buffer::replace_all and 2.41 GB through tree-sitter parsing for a single file. Changes: - Add FileLoadError::FileTooLarge variant with path, size, and limit fields - Add MAX_EDITOR_FILE_SIZE (50 MiB) constant in warp_files - FileModel::open(): check file size before read_to_string - FileModel::read_content_for_file(): check file size (used by discard) - FileModel::reload_file_paths(): skip oversized files during auto-reload - GlobalBufferModel::populate_buffer_with_read_content(): safety-net guard - GlobalBufferModel::apply_open_buffer_response(): guard remote buffers - Handle new FileTooLarge variant in read_binary_file_context match Sentry: https://sentry.io/organizations/warpdotdev/issues/7259255054/events/ddc1194e8a7f40a595f65a9b2378740b/ Co-Authored-By: Oz <oz-agent@warp.dev>
Contributor
|
Closing as duplicate of #12125 — both add a 50 MB FileTooLarge file size guard in FileModel::open. Keeping the newest PR. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Add a 50 MiB file-size guard to prevent memory spikes when opening oversized files in the editor.
Root cause:
GlobalBufferModel::populate_buffer_with_read_contentloaded arbitrarily large file content intoBuffer::replace_all, triggering multi-gigabyteSumTree<BufferText>::append_strallocations andSyntaxTreeState::parse_texttree-sitter parsing. The pprof heap profile from the Sentry event showed 8.07 GB (75.94%) through the Buffer path and 2.41 GB (22.73%) through tree-sitter, for a total footprint of ~20 GiB.Fix: There were no file-size guards anywhere in the file loading pipeline. This PR adds layered protection:
FileLoadError::FileTooLarge— new error variant with path, size, and limit fieldsMAX_EDITOR_FILE_SIZE(50 MiB) — constant inwarp_files, generous for source codeFileModel::open()— checks file size beforeread_to_string(initial load)FileModel::read_content_for_file()— checks size (used by discard/reload)FileModel::reload_file_paths()— skips oversized files during auto-reloadGlobalBufferModel::populate_buffer_with_read_content()— safety-net content-length guardGlobalBufferModel::apply_open_buffer_response()— guards remote buffer contentFileTooLargevariant in matchLinked Issue
Linear: APP-4519
Sentry issue: https://warpdotdev.sentry.io/issues/7259255054/
Sentry event: https://sentry.io/organizations/warpdotdev/issues/7259255054/events/ddc1194e8a7f40a595f65a9b2378740b/
The linked issue is labeled
ready-to-specorready-to-implement.Where appropriate, screenshots or a short video of the implementation are included below (especially for user-visible or UI changes).
Testing
Verified the code compiles with
cargo check --package warp --libVerified
cargo fmtandcargo clippypass with no warningsThe fix is defensive and minimal: each guard early-returns with
FailedToLoadwhich the existing UI already handlesI have manually tested my changes locally with
./script/runAgent Mode
Conversation: https://staging.warp.dev/conversation/9d0a6926-1b89-4d3b-a3ea-109d18764f73
Run: https://oz.staging.warp.dev/runs/019e45a6-a645-7d9b-9c8c-e72d1160d1f7
This PR was generated with Oz.